U.S. Pat. No. 6,792,545 discloses a system and method for performing secure investigations of networked devices over a computer network, the content of which is incorporated herein by reference. In such a system, there may be multiple examining machines concurrently conducting investigations of the network. Such concurrent investigations may result in duplication of some of the search results. Although deduplication mechanisms currently exist to eliminate redundant data within a single device, existing deduplication mechanisms are not scalable to multiple devices that collectively and concurrently need to identify and eliminate redundancies of data items in real time. Accordingly, what is needed is a real-time deduplication mechanism across a scalable environment where multiple machines can be concurrently collecting against the same set of deduplication data.